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Abstract. Davies and Wakerly show that Byzantine fault tolerance can 
be achieved by a cascade of broadcasts and middle value select functions. 
We present an extension of the Davies and Wakerly protocol, the unified 
protocol, and its proof of correctness. We prove that it satisfies validity 
and agreement properties for communication of exact values. We then 
introduce bounded communication error into the model. Inexact commu- 
nication is inherent for clock synchronization protocols. We prove that 
validity and agreement properties hold for inexact communication, and 
that exact communication is a special case. As a running example, we 
illustrate the unified protocol using the SPIDER family of fault-tolerant 
architectures. In particular we demonstrate that the SPIDER interactive 
consistency, distributed diagnosis, and clock synchronization protocols 
are instances of the unified protocol. 


Keywords: fault tolerance, protocol, SPIDER, Byzantine, reliability, Diagno- 
sis, Interactive Consistency. 


1 Introduction 

Safety-critical real-time applications rely on basic fault-tolerant services such 
as interactive consistency (IC), clock synchronization (CS), and distributed di- 
agnosis (DD, also called group membership). These services are usually ren- 
dered by distinct protocols that are designed, implemented, and validated sep- 
arately. Examples of systems that provide these services are SAFEbus [HD92], 
TTA [Kop97], and MAFT [KWFT88]. Rushby presents an overview of how sev- 
eral architectures realize these fundamental services [Rus03] . 

Davies and Wakerly, in their ground-breaking paper [DW78], observed that 
Byzantine fault tolerance can be achieved through a cascade of middle value 
select functions. This is true when exact values are communicated, such as the 
payload messages in IC or the accusations in DD. It is also true when inexact 
values are communicated. By inexact values we mean values that range over 
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the real numbers that may change by a bounded error during communication. 
Timing values for CS or analog sensor values are typical examples. Correct op- 
eration of a system crucially depends on both exact and inexact communication 
satisfying suitable validity and agreement properties. 

We introduce a generalization and extension of the Davies and Wakerly pro- 
tocol, which we call the unified protocol. Instances of this general protocol provide 
the core set of fault-tolerant services. We model the unified protocol formally 
and prove validity and agreement results for both exact and inexact data, under 
suitable fault assumptions. The exact case is precisely the inexact case with zero 
accumulated error. We then demonstrate how the unified protocol can be used as 
a basis for the IC, DD, and CS protocols for the SPIDER fault-tolerant architec- 
ture [MMTP02], We have verified the unified protocol using PVS [ORSvH95], a 
semi-automated theorem-proving system developed at SRI. The PVS proof files 
are available on the web [SPI] . 

The original contributions of this paper include a formally verified general- 
ization of the Davies and Wakerly protocol, adapted to exploit diagnostic infor- 
mation in the context of a hybrid fault model. In addition, we hope to rekindle 
interest in Davies and Wakerly’s results, which provide an effective approach for 
Byzantine fault tolerance for real-time embedded applications. 

The structure of this paper is as follows. Section 2 presents the unified pro- 
tocol. Section 3 presents the assumptions and requirements for the protocols 
described in this paper. Section 4 presents the analysis of the protocol for ex- 
act communication, and then illustrates how the SPIDER IC and DD protocols 
are instances of the unified protocol. Section 5 presents the analysis when the 
communication can introduce error, then demonstrates how the SPIDER CS 
protocol is an instance of the unified protocol. 

2 The Unified Protocol 

The unified protocol is a multiple stage protocol which is constructed from a 
single basic operation: a middle value select. In this section, we describe the 
middle value select function and then present the unified protocol using it. We 
conclude with a mapping of the unified protocol to the SPIDER fault-tolerant 
architecture. 

A distributed system is modeled as a graph with directed edges. Vertices 
are called nodes and directed edges are called links. We call s the source node, 
and d the destination node of the link (s,d). A communication stage is a set 
of source nodes, a set of destination nodes, and a set of of links between them. 
The absence of a link is modeled conservatively as a link fault. We allow both 
nodes and links to fail. However, we abstractly model link failures as failures of 
the source node [PMMG04]. 

2.1 Notation 

We use i or j to refer to an arbitrary stage and k to refer to the total number 
of stages. In the first stage of a fc-stage protocol, each member of the set N° of 
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nodes broadcasts to all members of the set TV 1 of nodes; in the second stage, each 
member of TV 1 broadcasts to all members of TV 2 , and so forth, up through the 
k th stage. Let us now fix an arbitrary stage i + 1 for 0 < i < k. The set of source 
nodes is TV*, and the set of destination nodes is TV* +1 . We use s, Si, S 2 , ■ ■ ■ € TV* 
to denote source nodes and d, d\,d- 2 , ■ ■ ■ € TV * +1 for destination nodes. When we 
refer to a node without refering to a communication stage, we use n G TV*. In 
the trivial example of a 0-stage protocol, no communication takes place and TV 0 
is the only set of nodes. 

Now we turn our attention to the values that are transfered at each stage. 
We model payload data using real numbers. We augment the set of reals with 
certain special values to indicate error conditions. Specifically we define a type 
T by 

T = {receive .error} U {source -error 1 \ i € N} U R. 

Let i>*(s) € T denote the value that s € TV* intends to broadcast in stage i + 1. 
After communication in this stage, each destination d has a vector of values 
v l d , such that v d (s ) is d’s estimate of v l (s). If the message that d receives from 
s is obviously incorrect (for example, it does not arrive within the expected 
window or fails a cyclical redundancy check), then v d (s) = receive -error . The 
value source -error 1 is a special message that is used to report the total absence 
of credible sources in stage i + 1. 

2.2 Middle Value Select 

The main computation during the execution of a single stage of the protocol is 
a middle value select voting algorithm. This algorithm chooses the middle value 
from the vector of received values, v d . For the data type T, we extend the natural 
order on the reals by the relations: 

— receive-error < source -error 0 , 

— source -error 1 < source-error if * < j, 

— source -error 1 < x for all i£l. 

Values from sources that are known to be faulty can be excluded from con- 
sideration. For this purpose, we define the filtered eligible sources, F d , to be the 
set of sources whose values are included in the vote computed by node d. If the 
cardinality of F d is even, any value between the two middle eligible values is an 
acceptable result, provided that all good nodes implement the same selection 
function. Let mvs(F d ,v d ) denote the middle value of the received values from 
the filtered eligible sources. 

2.3 Protocol 

The unified protocol is composed of a cascade of individual communication 
stages. A fc-stage protocol operates on the node sets TV°,...,TV fc . These sets 
may or may not be disjoint. For 0 < i < k, the algorithm for stage i + 1 is shown 
in Figure 1. Each destination node d maintains a set E d C TV* of eligible sources. 
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The set E d is based, in part, on d’s view of the failure status of the sources. 
Recognize that because of faults and errors during communication, v l d (s) may 
differ from u*(s). 


For stage i + 1, let s £ N l and d € N l+1 

Communication: Each source s broadcasts v l (s) to all destination nodes. 

For each destination d, v d (s) denotes the value received from source s. 
Computation: Each d computes 
1. I l d = {s | v d (s) = receive -error} 

2 .Fi = Ei\P d 

O V i+1(, n = f source-error \ if F d = 0, 

' \ mvs(F d ,v d ), otherwise. 


Fig. 1. Unified Protocol 


We assume that all correctly operating nodes share common knowledge of the 
communication schedule. In order to maintain integrity of the communication 
schedule, we require that correctly operating nodes be synchronized within a 
known precision. This synchrony provides a global time reference to manage 
the system’s time-triggered communication. Synchrony is maintained by a CS 
protocol. 

The protocol presented in Figure 1 generalizes the Davies and Wakerly (DW) 
protocol [DW78]. In the DW protocol, every stage has the same number of 
nodes. There is no such restriction on the unified protocol. Furthermore, the 
DW protocol does not use accumulated diagnostic information. At each stage, 
all nodes vote using identical sets of inputs. In the unified protocol, distinct nodes 
may compute the vote using nonintersecting vote sets. This capability enables 
the unified protocol to be analyzed using a weak hybrid fault assumption (see 
Section 3.4). 

2.4 Application: SPIDER 

The Scalable Processor-Independent Design for Electromagnetic Resilience (SPI- 
DER) is a family of general-purpose fault-tolerant architectures. The SPIDER is 
designed at NASA Langley Research Center to support laboratory investigations 
into various recovery strategies from transient failures caused by electromagnetic 
effects [MMTP02]. The unified protocol is used in SPIDER to implement the 
IC, CS, and DD protocols. One instance of the SPIDER architecture consists of 
several Processing Elements (PE) communicating over a Reliable Optical Bus 
(ROBUS). All application-level functions take place on the PEs. To the PEs, the 
ROBUS operates as a Time Division Multiple Access (TDMA) broadcast bus. 

The topology of the ROBUS is depicted in Figure 2. There are two types of 
nodes internal to the ROBUS. The Bus Interface Units (BIU) provide the only 
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interface to the PEs. The Redundancy Management Units (RMU) provide the 
necessary replication for fault tolerance. There is no direct link between any pair 
of BIUs nor any pair of RMUs. 



I l 


PEs 


BIUs 


RMUs 


Fig. 2. ROBUS architecture 


The primary uses of the unified protocol in the ROBUS are as 2- or 3- 
stage protocol instances. In a 2-stage instance, three sets of nodes are involved: 
./V^IV 1 and N 2 . For our subsequent discussions of the ROBUS protocols, N° 
corresponds to a subset of the BIUs, N 1 corresponds to the RMUs, and N 2 
corresponds to the BIUs. Communication is initiated from the BIUs (using in- 
formation from their attached PE) who send their values to the RMUs. The 
RMUs apply the middle value select and send their results back to the BIUs. 
The BIUs then apply another middle value select and forward the result to the 
PEs. Provided the system fault assumptions are maintained, the unified proto- 
col allows the ROBUS to provide strong guarantees about the timeliness and 
correctness of the communication between the various PEs. 

3 Protocol Analysis 

In this section, we explain the properties the unified protocol must satisfy: valid- 
ity and agreement. After a description of the fault model, i.e. , the covered kinds 
of faults, we define a fault assumption which constrains the number of faults of 
each kind. In the succeeding sections, we prove that the correctness conditions 
hold under this fault assumption. 

3.1 Correctness Conditions 

The unified protocol solves both the distributed consensus problem and the ap- 
proximate agreement problem, as defined in [Lyn96]. The IC and DD protocols 
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solve specific instances of the distributed consensus problem, and the CS pro- 
tocol solves a specific instance of the approximate agreement problem. Validity, 
agreement, and termination conditions are specified for each kind of problem. 
The unified protocol obviously terminates for a finite number of stages, so we 
do not formally state or prove this condition. 

Distributed Consensus Properties 

Validity If all nonfaulty processes start with the same initial value v £ V, 
then v is the only possible decision value for nonfaulty processes. 

Agreement No two nonfaulty processes decide on different values. 

Approximate Agreement Properties 

Validity Any decision value for a nonfaulty process is within the range of 
the initial values of the nonfaulty processes. 

Agreement The decision values of any pair of nonfaulty processes are 
within s of each other. 


3.2 Fault Classification 

Faults are classified according to the effect they have on the nodes of the sys- 
tem. We use a hybrid fault model from Thambidurai and Park [TP88] with one 
modification: benign nodes can sometimes behave as good nodes. The particular 
advantage of this modification is that many intermittent faults are now counted 
as benign, whence they are easy to mask. The nodes of the system are classified 
as follows: 

Good Each good node behaves according to specification; that is, it always 
sends valid messages. 

Benign Each benign faulty node either sends detectably incorrect messages to 
every receiver, or sends valid messages to every receiver. 

Symmetric A symmetric faulty node may send arbitrary messages, but each 
receiver receives the same message. 

Asymmetric An asymmetric (Byzantine) faulty node may send arbitrary mes- 
sages that may differ for the various receivers. 

A node that is not good is called faulty. A node is classified according to its worst 
error manifestation during the classification period. For example, it is possible 
for an asymmetric faulty node to behave in a manner that is observationally 
indistinguishable from a good node at times during this period. These classifica- 
tions form a “behavioral hierarchy” such that benign nodes can behave as if they 
are good; symmetric nodes can behave as if they are benign or good, etc. We 
let G, B , S, and A denote the sets of good, benign, symmetric, and asymmetric 
nodes, respectively. 

Good nodes always provide valid messages. Similarly, benign faulty nodes 
never provide misleading information. We define a set of nodes G, such that 
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the worst case error manifestation of a source in C is ommissive. 1 That is, a 
node in C can send a valid message or an obviously incorrect message, but can 
never communicate an invalid message. From the definitions above, we know 
that G U B C C. 

We attribute all faults to the communication, i.e., we assume that the process- 
ing of values by destination nodes is fault-free. We have described the rationale 
for this abstraction in [PMMG04], 

For the analyses presented in sections 4 and 5, we introduce the following 
definitions and supporting facts. Unless we explicitly state otherwise, we assume 

Cnivv®. 

<ax = rnax({V(n) | n € C D N‘ 1 }) 
fJnm = min({V(n) | n G C D N 1 }) 

Lemma 1. For all n € C C I N l , if v l min € R, then v l (n) € R . 

Lemma 2. For all s € C fl N l and d € C fl W , if u® • , t)b, € R, then 

K( S ) - vJ ( d )\ < m3X «ax - <i„. <ax - <i„) ■ 

3.3 Eligibility Assumptions 

In order to have a basis for agreement, we require that the sets of eligible sources 
differ only with respect to asymmetric sources. 

Let A be a family of sets of nodes. We say that X satisfies the Eligible Sources 
Property if all its members differ only in asymmetric nodes. 

Definition 1 (Eligible Sources Property (ESP)). 

ESP(X) = \/Xi,X 2 € X : n$A => (n G A x <=» n € X 2 ) . 

Let F d , F d be computed as in Figure 1. The families £ l , T l . F‘. (i < i < k of sets 
of eligible sources, ignored sources, and filtered eligible sources are respectively 
defined as follows: 

- I {E® | d G N i+1 } , 

- I 1 = {/* | d G N i+1 } , 

- r^{E\\F d \d&N^} . 

By definition, the filtered eligible sources inherit the Eligible Sources Prop- 
erty from their constituents: 

Lemma 3. If ESP (£*) and ESP{T), then ESP^ 1 ). 

We expect that £ l is derived from accumulated knowledge about the s £ N l , 
such that ESP (£*). In addition, we expect that the models of communication 
be analyzed to ensure ESP (I 1 ) for all i. The property ESP(T l ) can then be 
deduced by Lemma 3. 

1 Azadmanesh and Kieckhafer [AKOO] introduce the notion of a strictly ommissive 
asymmetric faulty node. In future work, we expect to extend our fault model to 
include this additional classification. 
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3.4 Fault Assumption 

Nodes can exhibit incorrect behavior; that is, they can fail. We require an inde- 
pendence of failure between nodes. Moreover we assume that a certain minimum 
number of nodes are operating correctly. Engineering design and analysis has to 
guarantee the satisfaction of these assumptions to a specified probability. A DD 
protocol provides mechanisms that can increase the probability that a sufficient 
number of nodes are operating correctly [LMK04] . 

Our fault assumption contains two clauses. Each clause is an assumption used 
to guarantee that validity and agreement hold. Agreement is established using 
two different fault assumptions: agreement propagation and agreement genera- 
tion. We name the clauses after the proofs in which they play a role. 

The first clause is called the Validity and Propagation Fault Assumption 
(VP FA). It states that for each destination and each stage between j and k, 
the majority of eligible, non-benign nodes are good. Formally, 

Definition 2 (Validity and Propagation Fault Assumption (VPFA)). 

VPFA(j, k) = Vz : j < i < k => Vd £ N i+1 : 2| G n E l 2 3 d \ > \E l d \ B\. 

The second clause is called the Agreement Generation Fault Assumption 
(AGFA) . It states that some stage between j and k is free of asymmetric, eligible 
nodes, and that the subsequent stages satisfy the VPFA. Formally, 

Definition 3 (Agreement Generation Fault Assumption (AGFA)). 

AGFA(j, k ) = 3i : j < i < kf\ESP(£ l )/\ VPFA(i+ 1, k)/\\/d £ N i+1 : \AnE l d \ = 0 . 

We have the following supporting lemmas: 

Lemma 4. For d £ N i+1 , if 2\G H E l d \ > \Ei\B\, then 2\C n Fj| > \F l d \. 

Lemma 5. For d\,d% £ N l+1 , if \ A fl E di \ = \A fl E d J = 0, ESP{£ 1 ), and 
ESP (J*), then F^ = F^. 

3.5 Application: SPIDER 

For the ROBUS architecture described in Section 2.4, we let N 2t denote the 
BIUs and N 2l+1 denote the RMUs, for any fc-stage SPIDER protocol and 0 < 
i < k. For k > 2, the SPIDER Maximum Fault Assumption is VPFA(j,j + k) A 
AGFA(j,j + k). This is equivalent to the following restatement of the SPIDER 
Maximum Fault Assumption [GM03]: 

1. 2|G n E r | > | E r \ B\ for all RMUs r, and 

2. 2|G n E b | > | E b \ B | for all BIUs 6, and 

3. | A fl E r | = 0 for all RMUs r, or | A fl E b \ = 0 for all BIUs h. 
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4 Exact Agreement 

In this section, we analyze the unified protocol assuming exact communications. 
We prove validity, agreement propagation and agreement generation under two 
assumptions on the communication. This framework is a special case of inex- 
act communication, addressed in the next section. However, this special case is 
simpler, so we present it first. 

4.1 A Model of Exact Communication 

For exact communication we assume that destinations receive exactly the mes- 
sages sent by good sources, that messages from benign faulty sources are either 
correct or ignored, and that all destinations receive exactly the same messages 
from non-asymmetric sources. 

More formally, we assume the following properties for the communication 
step in stage i + 1: 

Assumption 1 For all s £ C fl N l and d £ N l+1 , 

— s £ G and v d (s) = receive .error , or 

- < 4 ( s ) = ^ 0 )- 

Assumption 2 For all s £ N ' l \ A and di,d 2 £ N z+1 , v di (s) = v d2 (s). 

These assumptions define an implementation requirement for the commu- 
nication subsystem for any consensus protocol based on exact communication. 
The assumptions were constructed to ensure ESP (I 1 ). 

4.2 Exact Agreement Results 

In this section, we present the properties of the fc-stage protocol presented in 
Section 2.3 using the communication assumptions presented in Section 4.1. 

Theorem 1 (Upper Validity). If VPFA(j,j + k), then < u4 ax - 

Proof. By induction on k. 

The base case, k = 0, is trivial, so assume k > 0. By the induction hypothesis, 
we know that v 3 ^" 1 < v Lax- E remains to show that v^f a k < . Choose 

d £ C fl N 3+k such that v 3+k (d) = By VPFA(j,j + k), we know that 

2| G fl E 3 d +k ~ 1 \ > | E 3 d +k ~ 1 \ B\. By Lemma 4, we know that 2| C fl F 3 d +k ~ l \ > 
|Fj +fe_1 |. The pigeonhole principle ensures that there is an s £ (7nFj +fc_1 such 
that v 3+k (d) < v J d +k ~ 1 (s). Assumption 1 ensures that v 3 d +k ~ 1 (s) = v 3+k ~ 1 (s). 
The definition of v 3 ^^ 1 ensures that id +fc-1 (s) < □ 

Theorem 2 (Lower Validity). If VPFA(j,j + k), then w r ? nin < vf^ k . 

Proof. Similar to the proof of Theorem 1. □ 
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The following corollaries are direct consequences of Theorems 1 and 2. 


Corollary 1 

then id+k = v : 


(Consensus Validity). If VPFA(j,j + k) and 

,j +k 


— — 


= V. 


Corollary 2 (Master-Slave). If VPFA(j,j + k), v J m - m € K, and — v J min < 
A, then for all s £ C <1 , d £ C (~1 N^ +k we have |id(s) — v^ +k (d)\ < A . 


Corollary 3 (Agreement Propagation). If VPFA(j,j + k), v'F n £ 


and 


< A. 


then vtf k — uCt? < A. 


Corollary 3 ensures that agreement among receivers will be at least as good 
as the agreement among the sources. However, it does not provide assurance that 
exact agreement will ever be achieved. Specifically, the presence of an eligible 
asymmetric faulty node in every stage can prevent exact agreement. 

Theorem 3 (Agreement Generation). If AGFA(j,j + k) then vff a k = . 

Proof. By AGFA(j,j + k), there is a i < k such that VPFA(j + i + l,j + k), 
and | A fl E 3 d +l \ = 0, for all d £ CD N^ +l+1 . By Lemma 5, we know that 
F J d + l = F^ 1 = F, for di,d 2 £ Cf 1 N J+1+1 . Since F C N^ +l \ A, Assumption 2 
ensures that uj^*(s) = v J d + l (s) for all s £ F. Thus, vPfff" 1 = From 

Corollary 1, we get <+* = v \ (+ n fc . □ 


4.3 Application: SPIDER Interactive Consistency 

The SPIDER interactive consistency protocol [MMTP02] is an instance of the 
2-stage unified protocol. The properties we require of interactive consistency are 
the distributed consensus properties as defined in Section 3.1. 

Let s be some BIU that intends to send a value to all other BIUs. Next let v 2 
be computed using a 2-stage exchange with N° = {s}, N 1 the set of all R.MUs, 
and N 2 the set of all BIUs. The interactive consistency protocol for d £ N 2 is: 


ic ^ = J v2 ( d )’ if y2 ( d ) = ma J orit y( F d’ v d)’ 

[ no .majority , otherwise, 
where no-majority is a distinguished constant. 

Theorem 4 (IC validity). If s £ G and VPFA(Q. 2), then ic(d) = v(s). 

Proof. Since we have a singleton source set, u° lin = iy rlax = v(s). The result 
follows directly from Corollary 1 for k = 2. □ 

Theorem 5 (IC agreement). If AGFA(0,2), then ic{d\) = ic(d 2 ). 

Proof. The result follows from Theorem 3 for k = 2. □ 
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In addition, we are able to gather some diagnostic information about the 
source BIU, s. The following corollaries follow from Theorems 4 and 5, respec- 
tively. 

Corollary 4. If VPFA(Q. 2) and ic(d) = source -error 0 , then s £G. 

Corollary 5. If AGFA(0, 2) and ic(d) = no-majority, then s £ A. 


4.4 Application: SPIDER Distributed Diagnosis 

Distributed on-line diagnosis consists of two main parts. First, nodes accumulate 
evidence of faulty behavior by other nodes. Second, this local evidence must be 
reliably distributed to allow for global decisions. 

There are several mechanisms for accumulating evidence of faulty behavior. 
There are indirect mechanisms, such as those provided by Corollaries 4 and 5. 
There are also several direct accusation mechanisms. These include communica- 
tion resulting in receive -error and disagreement with results during an agreement 
propagation stage. 

We let D n (def ) € N represent node n’s accumulated evidence against defen- 
dant def . If D n (def ) = 0, then n has no recent evidence of faulty behavior by 
def . A larger D n (def) indicates more severe misbehavior by def. 

We require that a good node can never make a false accusation. Formally, 
if n € C and D n (def) > 0, then def £ G. The role of the distributed diagnosis 
protocol is to achieve global consensus from locally gathered accusations. Strictly 
speaking, SPIDER does not require a distributed diagnosis protocol. It is possible 
for the locally gathered accusations to satisfy the required assumptions. However, 
by periodically exchanging diagnostic information, we can remove accumulated 
disagreement caused by asymmetric faults. This can increase the probability that 
our fault assumptions are true, thus increasing the predicted reliability of the 
system [LMK04]. 

The SPIDER DD protocol is a 3-stage instance of the unified protocol. The 
first two stages are to assure agreement among the BIUs. The third stage is to 
propagate this consensus diagnostic information to the R.MUs. 

Let v°(b ) = Db(def), and v 2 and v 3 be computed using the 3-stage unified 
protocol. Thus, u° lax is the most severe correct local accusation against def and 
u)L n is the least severe accusation. 

Theorem 6 (DD Validity). If VPFA(0, 3), then for b € C fl N 2 , r € C fl N 3 , 

— If v 2 {b) > 0, then def ^ G. 

— Ifv 3 (r) > 0, then def ^ G. 

Proof. Both clauses are direct consequences of Theorems 1 and 2. □ 

Theorem 7 (DD Agreement). If AGFA(0,3) thenforb € CPiN 2 , r € CflV 3 , 
v 2 (b) = v 3 (r). 

Proof. Follows from Theorem 3 and Corollary 1. □ 
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The preceding results ensure consensus based on the BIUs local accusations 
against def. A similar protocol beginning with the RMUs ensures consensus 
based on the RMUs accusations against def. The maximum of the two results 
also satisfies validity and agreement. 


5 Approximate Agreement 

In this section we generalize the exact communication assumptions to accommo- 
date error introduced in the communication phase. The results in Section 4 are 
all special cases of the results introduced in this section. 

Analog information can be understood as a real valued, uniformly continu- 
ous function of time [Ros68]. Uniform continuity roughly means that the rate of 
change is bounded. For processing in a digital system, a digital approximation 
of the function value at a given moment is determined: the function is sampled. 
There are various sources of imprecision. For instance, the actual time of sam- 
pling may vary or the sampled value may be superposed with noise. The purpose 
of the inexact protocol is to reliably communicate values that may vary and may 
be further distorted during communication. 


5.1 A Model of Inexact Communication 

We model communication as in the exact case, but add terms representing the 
inherent imprecision of broadcasting inexact information. The error terms £i,£ u , 
and e are nonnegative reals. We define £ = £i + £ u . 

We assume that messages from good nodes are correctly received within a 
known error tolerance, that messages from benign faulty nodes are either ignored 
or are correctly received within a known tolerance, and that only asymmetric 
nodes may introduce disagreement beyond £ in the communication phase. We 
allow the communication error bounds, £i and £ u , to differ as the error may be 
biased. Formally, the assumptions for stage i+1 are: 

Assumption 3 For all s G C (1 N z and d G N t+1 : 

— s G and v d (s) = receive .error , 

— receive.error < v l d (s) = v l (s) < s ource. error 1 , or 

— v l (s) G R. and i>*(s) — £\ < v d (s ) < v l (s ) + £ u . 

Assumption 4 If s G N l \ A, then for d\,d 2 G N z+1 : 

— v ch( s ) = v d 2 ( s ) < source. error -1 , or 

— <4( s ).*4( s ) e R and l<4(s) ~ v d 2 ( s ) I < £ - 

When £ = £i = £ u = 0, Assumptions 3 and 4 reduce to Assumptions 1 and 2. 
Thus, exact communication is a special case of inexact communication. 
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5.2 Approximate Agreement Results 


The following results generalize the results from Section 4.2, by introducing the 
effects of bounded errors. By requiring v 3 min £ R for these results, we avoid 
the clutter of defining arithmetic involving source-error. These results can be 
extended to handle such special cases. 

Theorem 8 (Inexact Upper Validity). If VPFA(j, j + k) andv 3 min £ R, then 


v j+k < v j 

u max — ^max 


ks u . 


Proof. Similar to proof of Theorem 1. □ 

Theorem 9 (Inexact Lower Validity). If VPFA(j, j + k) and v J min E R, then 


v 


j 

min 


— ke i < V 


j+k 

min * 


Proof. Similar to proof of Theorem 1. □ 

Corollary 6 (Inexact Master-Slave). If VPFA(j,j + k), v 3 min £ R, and 
v max ~ v tiin — A then f or all s £ C C\ , d £ C fl Ni +k we have 

|id(s) — v 3+k (d ) | < A + max(fcei, ke u ). 


Proof. Follows directly from Lemma 2 and Theorems 8 and 9. □ 

Corollary 7 (Inexact Agreement Propagation). If VPFA(j,j + k), v 3 n - n £ 
R, and < ax - < in < A, then 


v 3 + k — v 3 + k < A 

max min — 


ke . 


Proof. From Theorems 8 and 9, we have — v 3 n kk < (fj^ax + tce u ) — i v min ~ 

ke i) < A + ke. □ 


Theorem 10 (Inexact Agreement Generation). If AGFA(j,j + k) and 

V min e R - then 


v 3 ~ kk — v 3 ~^ k 

^max c min 


< ke. 


Proof. Similar to proof of Theorem 3. 


□ 


5.3 Application: SPIDER synchronization protocol 

A clock is formalized as a function from clock time to real time. Clocks dis- 
tributed in a system need to be re-synchronized periodically in order to prevent 
them from drifting too far apart. The two goals of synchronization are: 

Accuracy All good clock readings are within a linear envelope of real time. 
Precision At all times, the clock times of all good clocks differ by a bounded 
amount. 
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Prior formal models of fault tolerant clock synchronization [SvH98,Min93,Sha92] 
have established a systematic way to derive accuracy and precision from the 
following properties: 

Accuracy Preservation The resynchronization time of a good clock is within 
the expected resynchronization times of good clocks, up to an error margin. 
Precision Enhancement If the skew of good clocks is within a known bound 
at the time of protocol execution, then all good clocks are synchronized to 
within a tighter skew after protocol execution 

Below we show how to prove accuracy preservation and precision enhance- 
ment, using validity and agreement properties of the unified protocol. The val- 
ues communicated during this protocol are estimates of the real time that nodes 
should reset their clocks for the next period. 

Let c n (T p+1 ) € R. denote the real time that node n expects to begin synchro- 
nization period p + 1. Let c' n (T p+1 ) denote the real time that node n actually 
begins period p- 1-1. Put another way, c n models n’s clock before resynchroniza- 
tion, and c' n models n’s clock after resynchronization. 

The SPIDER synchronization protocol is a 3-stage instance of the unified 
protocol. The BIUs are N° and N 2 , the RMUs are N 1 and N 3 . Let v°(bo) = 
Cb 0 {T p+1 ), for BIU bo £ N°. Then, for all b £ N 2 , r £ N 3 , define 

c' b (T p+1 ) = v 2 (b) 

c' r (T p+1 ) = i> 3 (r) 

The values £i and £ u bound the variation of clock readings caused by drift, 
jitter, and differences in communication delay. Let c m j n (p) and c max (p) denote 
the minimal and maximal values of all Cb(T p+1 ) such that Cb is a correct BIU 
clock at round p. 

Within the ROBUS, we are principally concerned with the accuracy of the 
BIUs, as these provide time references for the PEs. If needed, a similar argument 
can be used to bound the accuracy of the RMUs. 

Theorem 11 (BIU Accuracy Preservation). If VPFA(0,2) holds during 
synchronization period p, then for all good BIU clocks c' b : 

Cmin(p) - 2£l < c' b (T P+1 ) < C max (p) + 2£ u . 

Proof. Follows immediately from Theorems 8 and 9. □ 

Precision results are given for the set of BIUs, the set of RMUs, and between 
the BIUs and RMUs. This last result provides the skew bounds necessary to 
reliably communicate within the ROBUS. 

Theorem 12 (Precision Enhancement). If AGFA(0, 3) then 

1. \c bi (T p+1 ) — c b2 (T p+1 ) \ < 2s, 

2. K 1 (T p+1 ) - c' r2 (T p+1 )\ < 2s, 
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3. \c' b {T p+1 ) -c' r (TP +1 )\ < 2e + max(e 1 , e u ). 


Proof. Clauses 1 and 2 each follow from Theorem 10 (Clause 1 using AGFA(f), 2), 
and Clause 2 using A GFA(1, 3)). Clause 3 is a consequence of Clause 1 and 
Corollary 6. □ 


6 Concluding Remarks 

We introduce a formal model of an extension of the Davies and Wakerly protocol, 
called the unified protocol. We prove that under a weak hybrid fault assumption, 
the unified protocol satisfies validity and agreement, both for exact and inex- 
act communication. Three fundamental fault-tolerant protocols are shown to be 
instances of the unified protocol. 

With the unified protocol, the analysis of fault-tolerance properties can be 
restricted to one general protocol. In this way, the unified protocol provides a 
useful abstraction layer: the analysis of the fault tolerance is not complicated by 
specific concerns of individual protocols. For the SPIDER architecture, this has 
resulted in simpler specifications. This in turn yields a simpler implementation 
and more transparent treatment of the separate functions. Although we have 
not yet performed the analysis, we believe that the SPIDER transient recovery 
and restart protocols are also instances of the unified protocol. 

The unified protocol is flexible and can be adapted to other fault tolerant 
applications. In particular, it should be possible to adapt some of the arguments 
provided by Caspi and Salem [CS00] to bound the effects of computation error 
for locally computed control functions between communication stages. 

In addition, we expect that our results may be extended to analyze other ar- 
chitectures. Similar arguments may be constructed under weaker fault assump- 
tions. In particular, we intend to explore the benefits of extending our analysis 
to incorporate the strictly ommissive asymmetric classification introduced by 
Azadmanesh and Kieckhafer [AK00]. We also plan to explore a wider range of 
fault tolerant averaging functions within our PVS framework. Ultimately, we 
intend to provide a PVS library of reusable fault tolerance results. 
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